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ABSTRACT 



This paper applies the quality concept of continuous 
improvement to academic departments, very few of which engage in a continuous 
improvement (Cl) initiative. The goal of a Cl initiative is for a department 
to become self -regarding, self -monitoring , and self -correcting . Cl efforts 
begin with the development of a department -level assessment plan that 
identifies the mission of the department while being practical and 
principled. Steps to developing a departmental assessment plan are: (1) agree 

on the mission; (2) identify the scope of the plan; (3) create goals; (4) 
identify related processes and activities for each goal; (5) brainstorm 
appropriate measures; (6) evaluate and select measures; (7) identify 
appropriate assessment methods; (8) develop a plan for collecting data; (9) 
prioritize goals, set timelines, identify milestones; (10) implement 
assessment plans; (11) use data to improve processes; and (12) communicate 
results. A brief example illustrates the process and the considerations 
departments must recognize in the Cl process. (SLD) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



Department Level Assessment: 
Promoting Continuous 
Improvement. 

By Susan Hatfield 

DDE A Paper #35 

IDEA CENTER 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 

\ft i.Cdtm 



Oifice of Educational Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
> CENTER (ERIC) 

W This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

1 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



BEST COPY AVAILABLE 



2 



®@^(3[?C , [M)@[]D , Lj 1 [L@w@0 Z^ggOggDuQODQOg Pt?®[jLQ®ffi][IDgJ 

(^©dqC’Bdq©®©® □m[p[?®w@m@[]D'i) 

Susan R. Hatfield 
Winona State University 



©ejp^oi'meoi)!? [Le^eD AssessmeonS 



The purpose of this paper is to apply the quality concept of 
continuous improvement to academic departments. Most 
academic departments engage in periodic self-reviews as part 
of their university's three or five year program review process 
or in anticipation of an impending accreditation visit. These 
reviews generally provide a snapshot of the departments 
functioning at a particular point in time. Very few academic 
departments (except perhaps those with professional pro- 
grams) engage in anything which could remotely be called a 
Continuous Improvement (Cl) initiative. 

There are a number of reasons why Cl is not part of most 
departments' culture. Among these is the fact that Cl is usually 
an "add on" to a department and department chair's responsi- 
bilities, and not compensated in terms of release time or 
additional resources. Those chairs that have recognized the 
value of a departmental assessment plan often wind up writing 
it themselves, and as a result the plan has limited buy-in from 
departmental members whose support is needed for imple- 
mentation. Inherent in the Cl process is the fear of learning 
about oneself. Learning that "business as usual" is not as 
effective as had always been assumed can have political 
implications in terms of staffing and budgets. 

Still, becoming self-regarding by engaging in and being able 
to document continuous improvement on the department level 
is something accreditation agencies expect, legislatures 
demand, and the public applauds. Therefore, it is likely that Cl 
will become part of an increasing number of departments' 
cultures. 

The goal of a continuous improvement initiative is for a 
department to become self-regarding, self-monitoring and self- 
correcting. The result is that the department is confident of the 
quality of their graduates' knowledge and skills before the 
students enter the work force. Finding out how students have 
fared years after graduation isn't good enough. Continuous 
improvement means that the "correction cycle" is shorter 
because progress is constantly being monitored. Deficiencies 
are identified and are corrected while the student is still on- 
site, and corrections are made to department level processes 
to keep similar deficiencies from occurring. 




A department's continuous improvement efforts begin with the 
development of a department level assessment plan. There are 
a number of characteristics of good departmental level 
assessment plans. Specifically, a plan should be: 

o Principled. A good departmental assessment plan 
should be based on principles that are (ideally) defined 
at the university level. 

o Integrated. Department level assessment plans should be 
departmentally driven, but also tied to the university 
assessment initiatives and program review, carrying 
through themes related to outcomes in the major and 
general education. 

o Ongoing. Department level assessment should be part 
of the ongoing "business" of the department, not only a 
priority during program review cycles or prior to ac- 
creditation visits. 

o Comprehensive. Assessment activities should encompass 
students, faculty, and resources; inputs, process, and 
results. 

o Acculturated. Department level assessment can stand on 
its own, but to be optimally effective department level 
efforts need to be a valued and supported part of the 
university's culture. 

o Implemented gradually. Assessment needs to become 
part of the culture slowly, implemented in carefully 
orchestrated steps over time. 

o Practical. To be truly useful, department level assessment 
must stay on a practical level with obvious implications to 
faculty and students. 

o Self-Renewing. Assessment data and information must 
feed back into the system, both on the university level 
and departmental level. 

A department level assessment plan should identify the mission 
of the department, goals related to that mission, activities or 
processes supporting the achievement of the goal, and a 
number of measures which, when taken together, provide an 
indication of the degree to which the goal is being achieved. 
Implementing the plan requires the collecting, analyzing and 
benchmarking of data, revision of processes, and communica- 
tion of results. 
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Before a department can develop an assessment plan, it is 
helpful to understand what a department level assessment plan 
looks like. Essentially, a good departmental assessment plan is 
one that is workable and one that will yield information that 
will help a department learn about and improve itself. Having 
an elaborate plan that is paralyzing in its complexity will almost 
assure that the assessment plan will never be implemented. 

A good department level assessment plan is one that identifies 
a limited number of goals to which the department members 
have agreed and that have the potential to make a difference 
in how the department functions. 

Mission 
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Basically, the model plan has a limited number of goals, 
supported by a number of departmental activities or processes 
designed to support the achievement of the goal. Multiple 
measures taken together provide an indicator of the degree to 
which the goal is being achieved. 
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There are twelve steps to the development and implementation 
of a department level assessment plan. 

1 . Agree on the mission 

2. Identify the scope of the plan 

3. Create goals 

4. Identify related processes and activities for each goal 

5. Brainstorm appropriate measures 

6. Evaluate and select measures 

7. Identify appropriate assessment methods 

8. Develop a plan for collecting data 

9. Prioritize goals, set timelines, identify milestones 

1 0. Implement assessment plan 

1 1 . Use data to improve processes 

1 2. Communicate results 

1 . Agree on the mission 

This first step is much easier said than accomplished. The 
problem is that everyone has their own idea of the character- 
istics of a mission statement. For some people, a mission is 
essentially a slogan. For others, it is a statement of core 
values. Some argue that it should be a long statement, others 
Q that it should be no longer than a sentence or two. Still 
R I ( insist on bullets. 



It is possible for departments to spend a number of months on 
this step, and then never go any further because they are so 
tired or the process. Time will be better spent talking about 
what it is that the department really wants to accomplish and 
making sure that this is tied to the mission of the university. A 
minimum goal is for a "conceptual convergence" — agreement 
among department members as to (1 ) who it wants to serve, 
(2) in what ways, and (3) with what result. Agreeing on these 
three questions will provide a basis for continuing, if a formal 
statement can be crafted that everyone agrees upon, all the 
better. But a conceptual agreement is sufficient for proceeding 
to Step 2. 

2. Identify the scope of the plan 

One of the first tasks for a department, after determining the 
mission, is to determine the scope of the plan itself. What 
happens in a department can be understood by thinking of it 
as a system consisting of inputs (what goes into the depart- 
ment), processes (what happens while in the system), results 
(the outcomes), and feedback (how the results transform inputs 
and/or processes). A broad plan might consider goals in all 
of the categories: inputs; processes; results; and feedback for 
both students and faculty. A narrow plan might focus on 
limited elements (e. g., processes and results Tor either stu- 
dents or faculty). 

The cycle of department level continuous improvement starts 
with a set of clear departmental goals that encompasses not 
only the expected results (e. g., student learning, student 
satisfaction, faculty research), but also the related inputs 
(e. g., equipment, facilities, personnel) and, most importantly, 
relevant processes (e. g., curriculum, instruction, advising). 
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The goal of creating a departmental assessment plan, is to 
facilitate continuous process level improvement. In order to do 
that, the plan needs to be workable. A limited number of 
meaningful goals will be more beneficial to a department than a 
complex plan that is gathering dust on an administrative shelf. 
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3. Create goals 

Having identified the scope of the plan, the next step is to 
create goals within the scope of the plan. If the department 
has decided to focus on students' academic success, the goals 
would probably be related to academic processes which 
promote student learning (curriculum, teaching, advising), as 
well as goals which identify the intended result (knowledge 
and skills). 

4. Identify related processes and activities for each goal 

Each departmental goal should be supported by an identified 
list of processes ana activities that support the achievement of 
the goal. Unless specific processes or activities that support 
the goal can be identified, it is unlikely that the goal will be 
achieved in any kind of systematic manner. In such a case, the 
department needs to design activities specifically to encourage 
the achievement of the goal. 

5. Brainstorm appropriate measures 

At this point the question becomes "How can a department 
know to what degree they are achieving the goal?" There are 
a number of ways to do this. Direct measures are those that 
come from student interviews or questionnaires, or from 
directly observing student behavior. Indirect measures come 
from artifacts or interpretation — where the subjects are not 
aware that they are being studied. The resulting data can be 
either qualitative or quantitative. Qualitative data tends to 
yield a rich data set, but can be hard to interpret. Quantitative 
data such as percentages and ratios allow for easier interpre- 
tation, but at the cost of neglecting some of the finer points of 
the data. A combination of both will be most helpful. Quanti- 
tative data will be easier to track longitudinally, while qualita- 
tive data will provide a context for interpreting the quantitative 
measures. 

The idea is to brainstorm as many — and as creative — 
measures as possible. 

6. Evaluate and select measures 

Ideally, departments should select only a limited number of 
measures for each goal. There are a number of criteria which 
are useful in this process. 

o Is the measure important as opposed to merely interest- 
ing? Most data are interesting, especially to assessment 
people. But not all of it is important. The trick is to 
determine what is the most credible, valid way to make 
inferences about the progress on this goal. 

o Does the required information exist or will a new 

rocess be required to collect the data? A department 
as both limited time and resources. Best possible use of 
both need to be made. Selecting measures that will 
require the creation of elaborate data collection methods 
should be undertaken only after a serious consideration 
of costs and benefits. Perhaps the best place to start is 
with data that might already exist. The Institutional 
Research Office, Hie Registrar's Office, the Placement 
Office, and the Alumni Office might all have information 
useful to a department. Finding out what information 
these offices routinely collect and if data from a specific 
department or group of students can be pulled from their 
databases might save considerable effort. 
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o Is the measure easily understood? Does it have high 
"face validity"? Does it lend itself to unambiguous 
interpretation? Anyone who has collected data has been 
faced with the situation in which the data are 
uninterpretable. Think about whether or not alternative 
interpretations/explanations of the data are possible 
before selecting the measure. 

o Are fluctuations in the measure a result primarily of the 
department's/unit's actions or are extraneous factors 
more influential? This is a simple one. If the department 
does not directly control the input or process responsible 
for the outcome, it should not be selected. 

o Is it possible to measure the outcome in a way that 
preserves its integrity? Some outcomes might be very 
important, but credible measures are difficult to obtain. It 
is important to know why students leave a university, but 
student responses to queries about this are strongly 
influenced by social desirability pressures. For a fot of 
students, it is easier to shade the truth rather than admit 
that they are homesick, miss their girlfriend, or are 
flunking out. 

o Does the measure change over time? Some measures 
are too stable over time to be sensitive to growth pro- 
cesses. Unfortunately, it is difficult to identify these 
measures until they have been collected for a couple of 
years. 

o Does the measure signify achievement of a goal or 
merely the success of a related activity? This is a big 
issue. Remember, the idea is to measure the progress 
toward the goal . So if the goal is to increase 
multicultural sensitivity on campus, counting the number 
of students who attend a lyceum on the topic is more a 
measure of the success of the activity (and a specious 
one at that, since many students are bribed to attend 
campus events with extra credit), than of increased 
sensitivity on the campus. 

o Will the measure provide information which can be used 
for improvement? For each measure, ask: "What can 
be learned from this?" and "How can this information be 
used to develop improvement strategies?" If the measure 
does not point to any possible process improvement, it 
may not make sense to collect it. 

7. Identify appropriate assessment methods 

Once the department has an idea of what their goals are and 
what constitutes success, the next step is to figure out how to 
assess the goals. There are a number of methods for both 
process and outcome assessment, including: 

Locally prepared exams 

National exams (licensure, graduate school, professional 
society, board exams, etc.) 

Oral exams 

Surveys (locally developed, nationally normed, students, 
alumni, employers) 

Journals/application papers 
Inventories 
Database elements 
Transcript analysis 
Curriculum or Skills Mapping 
Portfolios 

Case studies and analyses 
Benchmarking 

Interviews (individual, focus groups) 
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offerings each semester. A departmental e-mail list of faculty 
and students was created and utilized to send out relevant 
academic information to students, including information 

f )ertaining to the changes in the advising process, the reasons 
or the change, and the goals of the new processes. 

A departmental Advising Day was designated prior to prereg- 
istration for the following semester. This day replaced that 
usual two-week-long preregistration advising period when 
students would meet with faculty individually during office 
hours. During Advising Day, all faculty were available in a 
centralized location to meet with students. Handouts were 
prepared and distributed to each student identifying deadline 
dates and other important information. At least tour faculty 
were available in the advising room throughout the day, with 
the rationale that all student questions would be able to be 
answered by someone in the room at the time. Faculty were 
also able to confer on issues that were presented and able to 
solve problems and make decisions at that time in that room. 

Advising Day allowed faculty to keep track of students partici- 
pating in the advising process, update the departmental files 
on individual students, and also collect data on student study 
habits, satisfaction, etc. Faculty also worked with students to 
plan out their academic careers, using the three year schedule. 
Following advising, the three year schedule planning sheets 
allowed the department chair to recognize potential enrollment 
"bubbles" and add additional sections of needed courses 
during future semesters' schedules to accommodate enrollment 
patterns. 

Results . Around eighty percent of departmental majors 
regularly participate in Advising Day. Ninety percent of 
graduation applications are approved the first time they are 
submitted. Students report high levels of satisfaction with 
the advising process and amount of information available 
to them. The department chairperson and faculty have 
noted significantly fewer student issues being brought to the 
department. Overall, the adjustments to the advising 
process seem to be working. Feedback from students and 
Faculty continues to be collected in an attempt to monitor the 
advising process and revise it as necessary. 

C@(n)SDdle[r@i1tD®[nis 

There are a number of considerations (and cautions) for 
departments undertaking a continuous improvement initiative. 

o Keep th< 
number 

o Keep the university's mission and goals and any pro- 
ram review requirements in mina so that each initiative 
oes not require a separate series of activities. Ideally, a 
department assessment plan should be linked so that any 
reasonable request for data (which presupposes that 
requests for data will be reasonable) can be handled 
easily. 

o Concentrate on only one or two goals at a time. Perhaps 
it would be helpful to pick one or two assessment themes 
for the year. 

o Remember that different goals may be in different stages 
of development. In some cases, the department has aata 
and can start revising processes. In other cases, activi- 
ties need to be designed to facilitate the achievement of 
the goal. 
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o For everyone's sanity, the process needs to be ongoing. 
Bursts of assessment activities are not likely to yield 
genuinely useful information and are stressful. 

o Expect to "trickle up" continuous improvement activities 
through the curriculum and student body. It is simply 
easier to work with cohorts of new students than it is to 
change the rules on upper division students. 

o Feed back information into the system. Let faculty, 
students, and administration know what has been 
learned and how that knowledge has changed operating 
procedures. 

o Take the opportunity to engage in benchmarking when- 
ever possible. There seems to be a feeling that everyone 
is supposed to figure out assessment by themselves. 

There isn't much sense to having everyone reinvent the 
wheel. Take the opportunities to learn from other 
departments who are ahead and offer assistance to 
those who are behind. 

Engaging in Continuous Improvement activities should be part 
of every department's "business as usual." Implementing a 
departmental assessment plan in a systematic matter allows 
departments to continually assess the knowledge, skills, and 
abilities of their graduates as well as the quality of the internal 
processes that produce those graduates. This information is 
essential for academic departments to grow and adapt to 
changes both inside and outside of the university walls. 
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Performance/Presentation (exhibitions, projects, research) 
Capstone experience 
Theses, Internships 
Simulations & Role Play 

Each of these methods has advantages and drawbacks in 
terms of implementation, expense, and analysis. In some 
cases, assessment can be embedded into a capstone 
experience such as an internship or senior thesis. Student 
portfolios might also offer opportunities for both process 
and outcome level assessment. It is desirable to design an 
assessment method that will provide measures of a number 
of different goals. 

8. Develop a plan for collecting data 

At this point the department has identified goals along with 
processes and activities, brainstormed measures for each goal, 
icked a small number of the best measures, and identified the 
est method to collect the information. Now it is time to figure 
out how the information is going to be obtained. There are 
several important considerations: 

o Why is this important? This is one last chance to 
weed out any bad choices that have been made about 
measures. 

o From whom can we get this information? Identify the 
population to be assessed. Think about the kind of 
information being sought and who would have that 
information. Remembering the bias in self report data, it 
is important to consider other assessment methods. It 
might be useful to consider "stealth assessment" — 
records that exist in places like the Registrar's Office or 
Financial Aid Office that might provide accurate infor- 
mation free from the drawbacks of self-reports. 

o How can it be collected? Details are important. For 
instance, if surveys have been selected as a means of 
assessing student satisfaction, consideration needs to be 
given to now they will be administered — telephone, mail, 
WWW, e-mail, paper and pencil? The key is to balance 
costs and rewards such as response rates, credibility, 
and confidentiality. 

o Can the data collection serve more than one purpose? 
If the department is interested in assessing both critical 
thinking and writing skills, it might be possible to 
design a writing exercise that requires the demonstra- 
tion of both. 

o When is the best time to collect the data? Timing is 
critical. Consider the rhythm of the academic year. 
Anyone who has distributed course evaluations during 
the final exam period has learned this fundamental 
lesson. 

o How can the data be evaluated? Will the data have 
inherent meaning, or will the meaning come from 
triangulation with other data, benchmarks, or a 
baseline? 

o How can the data be used? If it is not clear how the 
information will be useful, if it doesn't point to some 
process that can be improved, then it is not a good 
measure. 

o How can the data be reported? Qualitative data is 
harder to collect, analyze, and report than quantitative 
data. On the other hand, qualitative data adds a 
owerful, often persuasive dimension to quantitative 



9. Prioritize goals, set timelines, identify milestones 

Ideally, a department should prioritize goals after having 
worked through the first eight steps with each of the goals that 
have been identified. Having done this, the department has a 
full understanding on what working on each goal entails and 
can make a responsible decision about which goals to work 
on first. The decision is based upon information as opposed to 
intuition. 

But there's a balance that needs to be considered. If develop- 
ing all of the department goals up to this point means that the 
department loses interest in assessment, then it makes sense to 
move ahead sooner. Getting started on something is better 
than being noble in principle but lost in detail. 

It is recommended that a department should develop fully 
(Steps 1-8) four to six department goals, and select two or 
three to work on each year. Be realistic. Two or three goals 
will keep any department busy. 

It is important to note that progress on specific goals will 
proceed at different rates. For instance, a department may be 
working on three different goals during the same year — 
collecting data on a goal for which there are already sound 
processes, developing activities in support of a second goal, 
and revising the activities for a third goal on the basis of data 
that have already been analyzed. 

Regardless of how many goals the department is working 
on during a given year, each should have a timeline which 
identifies outcomes to be achieved by the end of the aca- 
demic year. 

1 0. Implement assessment plan & 1 1 . Use data to improve 
processes 

Continuous Improvement on the departmental level occurs 
when department level data are systematically collected and 
evaluated on a regular basis, ana adjustments in inputs and 
processes are maae as the result of the data analysis. 

During the first year of the continuous improvement cycle, data 
are collected on specific measures. Such data constitute the 
baseline — a "business as usual" benchmark. In other words, 
this is the outcome that is obtained from current procedures 
with no special interventions. The department needs to decide 
whether or not the baseline measure indicates an acceptable 
level of achievement for the goal. Different types of goals will 
have different acceptable thresholds. For instance, it is 
probably unrealistic to expect that more than 80% percent of 
students will report "high" or "very high" levels of satisfaction 
with any particular departmental procedure, but it may be 
unacceptable if only 70% of students perform successfully on a 
writing assessment. The definition of acceptability or 
unacceptability is dependent on the importance of the goal, on 
the type of measure (objective or subjective), and on available 
benchmarks either within the university or externally. 

If the measure identifies what is deemed to be an acceptable 
level of achievement, the department can be comfortable 
maintaining its current practices and move on to other depart- 
ment goals, with the intent of re-monitoring this specific goal in 
another couple of years. 
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If the measure indicates an unacceptable level of performance, 
the department needs to examine the processes/activities to 
diagnose the problem. For instance, if only 70% of graduat- 
ing students in a particular department are able to write in a 
manner expected by department members of college gradu- 
ates (as evidenced by an analysis of student writing portfolios 
and student performance on a department-wide writing 
exam), then something appears to be wrong with the pro- 
cesses intended to develop writing skills. Questions that might 
arise include: Are there enough writing experiences in the 
major sequence? Are faculty providing nelpful feedback to 
students on their writing? Have students completed required 
writing courses (general education) prior to starting the 
major? Using the answers to these questions, department 
faculty can begin to adjust activities/processes with the goal 
of improving students' writing performance in subsequent 
years. In the example presented, faculty might incorporate 
more writing experiences into their courses across the curricu- 
lum sequence, require peer feedback on written assignments 
before assignments are turned in to faculty, have students turn 
in outlines and drafts throughout the semester, have major 
written assignments due three weeks before the end of the 
semester to allow for faculty feedback and revision by stu- 
dents, require students to complete the general education 
writing courses prior to declaring a major, or require those 
with marginal writing skills to spend time in the campus' 
writing center. Such interventions should be evaluated in 
terms of practicality and cost, and those with the most poten- 
tial implemented. 



During subsequent years, the same measures are 
collected and compared against the baseline. New 
activities continue to be developed and revised as 
necessary based upon the data. Once acceptable 
and stable measure is obtained, the department can 
begin to monitor that goal less often and instead focus 
on other priorities. 

12. Communicate results 

Departments often overlook this last step. To be 
effective, assessment must be a transparent process — 
after the assessment plan is implemented, chanqes 
made as a result of the assessment process need to be 
made public. This is especially critical if the assess- 
ment plan requires the participation of students and 
faculty (and it is hard to imagine a plan that would 
not). Nothing will kill motivation to participate in 
assessment quicker than to fill out seemingly endless 
questionnaires and surveys and never see anything 
changed as a result of the feedback provided. 



A Metf Ex 

Problem : A department chairperson and departmen- 
tal faculty at a 7,000-student midwestern university 
noticed that more and more of their departmental 
meeting time was being taken up with individual 
student issues such as course substitutions and 
requests for waiver of GPA requirements along with 
students whose applications for graduation were 
being denied by the Registrar's Office because of 
degree requirements that had not been met. Addition- 
ally, faculty were complaining that a number of 
students were taking upper division courses without 
having completed the prerequisites. It also appeared from the 
nature of the student requests that they had received incorrect 
information from faculty. All of these issues seemed to involve 
advising. 

Action : Faculty noted from their list of advisees the number of 
students who actually participated in formal advising, either 
during the preregistration period or throughout the academic 
year. The percentage of students varied widely among faculty 
members. Faculty also surveyed graduating seniors to get an 
indication of the percentage of students whose applications 
for graduation were approved without revisions by the 
Registrar. Neither of these percentages were satisfactory to 
faculty. The data made it clear that the present advising 
system was not working to the satisfaction of faculty nor to the 
advantage of students. Faculty decided to redesign their 
departmental processes surrounding advising. The redesign 
essentially revolved around two issues: getting information 
out to students, and making student and university information 
available to faculty. 

A departmental database was created that included every 
student major. This database allowed faculty to track a 
student's progress through the curriculum and included current 
local address, e-mail address, and phone number. Addition- 
ally, a department file was created for each student which 
contained similar information. A three year course schedule 
was published and widely circulated which listed the class 
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